Systems, apparatuses, and methods for providing a ranking based recommendation

ABSTRACT

One or more of the systems, apparatuses, or methods discussed herein can include a quality score for a plurality of item listings or collections of item listings. Data sparseness can be avoided, as the quality score is based on inherent properties of the listing. An item listing can be recommended to a user based on the quality score. In one or more embodiments, a method can include determining a plurality of quality scores including a quality score for each of a plurality of item listings or a plurality of collections of item listings, the quality scores determined independent of a user&#39;s attributes and independent of the user&#39;s contextual information, the contextual information corresponding to details of the user&#39;s access to a website, and recommending an item listing or collection of item listings to a user based on the quality scores and the contextual information.

RELATED APPLICATION

This application claims the benefit of priority under 35 U.S.C. Section119(e) to U.S. Provisional Patent Application Ser. No. 62/043,064,entitled “SYSTEMS, APPARATUSES, AND METHODS FOR PROVIDING A RANKINGBASED RECOMMENDATION,” filed on Aug. 28, 2014, which is incorporatedherein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to providing a recommendationto a user and more specifically to providing a recommendation using aquality score and/or making a recommendation context-aware.

BACKGROUND

Recommendation systems (RSs) have been in the e-Commerce industry formore than a decade. Conventional RSs, such as Collaborative Filtering(CF) and Content Based (CB) based RSs include many drawbacks. CF and CBalone do not incorporate contextual information. For betterpersonalization and user experience Context Aware Recommendation Systems(CARSs) were introduced. Most of the existing CARSs are built on top ofconventional CF or CB models. The CARSs have drawbacks as well.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments of the present disclosure are illustrated by way ofexample and not limitation in the figures of the accompanying drawings,in which like reference numbers indicate similar elements.

FIG. 1 illustrates, by way of example, a network diagram depicting aclient-server system, within which one or more embodiments may bedeployed.

FIG. 2 illustrates, by way of example, a block diagram of marketplaceand payment applications and that, in one or more embodiments, areprovided as part of application server(s) 118 in the networked system.

FIG. 3 illustrates, by way of example, a block diagram of an embodimentof a recommendation system that, in one or more embodiments, is providedas part of the application server(s) 118 and/or the networked system.

FIG. 4 illustrates, by way of example, a flow diagram of an embodimentof a method of contextual pre-filtering.

FIG. 5 illustrates, by way of example, a flow diagram of an embodimentof a method of contextual post-filtering.

FIG. 6 illustrates, by way of example, a flow diagram of an embodimentof a method of contextual modeling.

FIG. 7 illustrates, by way of example, a table of simulation results ofcontextual modeling using attributes at a collections level.

FIG. 8 illustrates, by way of example, a table of simulation results ofcontextual modeling using attributes at a session level.

FIG. 9 illustrates, by way of example, a table of simulation resultscomparing contextual pre-filtering to contextual post-filtering.

FIG. 10 illustrates, by way of example, a flow diagram of an embodimentof a method of providing a recommendation.

FIG. 11 illustrates, by way of example, a block diagram of an embodimentof a mobile device.

FIG. 12 illustrates, by way of example, an embodiment of a machine thatmay be used to perform one or more techniques (e.g., methods) discussedherein.

DETAILED DESCRIPTION

The description that follows includes illustrative systems, methods,techniques, instruction sequences, and computing machine programproducts that embody illustrative embodiments. In the followingdescription, for purposes of explanation, numerous specific details areset forth in order to provide an understanding of various embodiments ofthe subject matter. It will be evident, however, at least to thoseskilled in the art, that embodiments of the subject matter may bepracticed without one or more these specific details.

In one or more embodiments, a personalized recommendation is provided toa user through the use of various techniques. The personalizedrecommendation can be provided using a quality score ranking techniqueand/or using context information associated with a user's access to awebsite to filter item listings (e.g., listings of individual itemlistings or collections of item listings) prior to presenting the itemlisting to a user. The item listings can include one or more productsfor sale, a service offering, or an advertisement for a product orservice, among other item listings.

RSs have become important tools in personalized marketing. The existingsystems are usually based on CF, CB, or a combination of both (Hybrid).Some RSs integrate machine learning techniques with hybrid systems inthe hopes of avoiding drawbacks of CF/CB systems. Such machine learningtechniques assist in making the recommendation more customized, butrequire an increase in computer resources and increase an amount of timeit takes to provide a recommendation to a user. As used herein“recommending” or “recommendation” mean displaying an item listing to auser, such as under a heading indicating that the user might beinterested in the item listing.

In CF, recommendations are determined using techniques that involvecollaboration among multiple agents. These agents can be users. Theagents rate some item listing and then, based on some similarityfunction that determines a similarity between agents, recommendationsare made to similar users (e.g., users with some characteristic, such asa demographic, a preference, purchase history, or other characteristicin common). In CB, recommendations are made using the content of itemlistings by providing a recommendation for an item listing similar to anitem listing the agent viewed or an item listing the agent purchased,for example, and are independent of an agent's opinion(s) (e.g.,rating(s) of the item listings/item listings as in CF RSs).

CF and CB recommendation methods have many drawbacks. CF and CBrecommendation methods do not incorporate the context of the agentsvisit to an item listing. Context can be defined as the state of a user.Context information can be used in different ways with respect to arecommendation. Context information is different from user attributesthat are inherent to a user. User attributes can include userpreferences, demographics, purchase history, or the like.

The following are some limitations and challenges with includingpersonalization in CF or CB RSs. Both CF and CB have data sparsenesschallenges. Using CF, few data points per user leads to very fewrecommendations. Using both CF and CB, when there is no data for a user,new item listings are not recommended well (a so called “Cold Start”).In CF, there are many item listings to choose from and information maynot be available for a sufficient number of users, making therecommendation not very personalized. Some methods have been developedto deal with sparseness. Sparseness can be dealt with using singularvalue decomposition, matrix factorization, or clustering, each of whichis computationally very expensive, leads to information loss, andproduces bad recommendations for niche categories of item listings orcollections, and/or does not recommend a niche category at all. A badrecommendation can include an irrelevant recommendation to the user or arecommendation that is repugnant to a user, among other badrecommendations.

Other drawbacks of both CF and CB include similarity computations beingtime consuming, O(n²) with an n sized sample, CF tends to becomepopularity biased, there is a problem as to what to recommend with newitem listings using CF, and an absence of context leads to badrecommendations if two or more different users use an account. Furtherdrawbacks include a difficulty in incorporating ratings and qualityjudgments of other users using CB, diversity is difficult to implementusing CB, and there is limited content information, so discriminationamong item listings is difficult if not impossible in CB.

In general, both CF and CB recommendations use item listing-levellisting characteristics or an agent's historical information or ratingsfor recommending additional item listings to a user. Contextualfeatures, in contrast, are specific to a session or the context inaction of the agent. For example, a contextual feature can include whatkind of device is being used currently to access content (e.g., desktop,mobile/tablet), what time the access is occurring (e.g., aweekday/weekend/day time/nighttime), etc. Contexts are dynamic, and theycan change frequently with respect to the agent. Contextual informationcan change the way item listings are recommended as compared to the wayitem listings are recommended in a CF and a CB system.

The subject matter discussed herein can help improve over the prior RSs.The subject matter is applicable to virtually any RS and recommendationmethod. The subject matter discussed herein can provide a recommendationin an RS that uses collections of item listings. A collection of itemlistings is a group of item listings that are grouped together based ona common theme. The collection of item listings can be treated as asingle item listing in that it the item listing collection can receive asingle quality score based on one or more item listings of thecollection. The discussion of embodiments of RSs begins with adiscussion of a networked system in which an RS may be deployed.

FIG. 1 illustrates, by way of example, a block diagram of an embodimentof a network diagram depicting a client-server system 100, within whichone or more RS embodiments may be deployed. A networked system 102, inthe example form of a network-based marketplace or publication system,provides server-side functionality, via a network 104 (e.g., theInternet or a Wide Area Network (WAN)), to one or more clients. FIG. 1illustrates, for example, a web client 106 (e.g., a browser, such as theInternet Explorer browser developed by Microsoft Corporation of Redmond,Wash. State) and a programmatic client 108 executing on respectivedevices 110 and 112.

An Application Program Interface (API) server 114 and a web server 116are coupled to, and provide programmatic and web interfaces respectivelyto, one or more application servers 118. The application servers 118host one or more marketplace applications 120 and payment applications122. The application servers 118 are, in turn, shown to be coupled toone or more database servers 124 that facilitate access to one or moredatabases 126.

The databases 126 can include information stored thereon regarding itemlistings that can be recommended to a user. The item listings caninclude tags, such as can include a quality score tag that indicates aquality score of the item listing, a contextual information tag thatindicates contextual information that is consistent with the itemlisting, or other tag. The database 126 can include informationregarding item listings to be recommended to a user that arepre-computed offline (e.g., when the user is not online).

The marketplace applications 120 may provide a number of marketplacefunctions and services to users who access the networked system 102. Thepayment applications 122 may likewise provide a number of paymentservices and functions to users. The payment applications 122 may allowusers to accumulate value (e.g., in a commercial currency, such as theU.S. dollar, or a proprietary currency, such as “points”) in accounts,and then later to redeem the accumulated value for products (e.g., goodsor services) that are made available via the marketplace applications120. While the marketplace and payment applications 120 and 122 areshown in FIG. 1 to both form part of the networked system 102, it willbe appreciated that, in alternative embodiments, the paymentapplications 122 may form part of a payment service that is separate anddistinct from the networked system 102.

Further, while the system 100 shown in FIG. 1 employs a client-serverarchitecture, the embodiments are, of course, not limited to such anarchitecture, and could equally well find application in a distributedarchitecture, or a peer-to-peer architecture system, for example. Thevarious marketplace and payment applications 120 and 122 can also beimplemented as standalone software programs, which do not necessarilyhave networking capabilities.

The web client 106 accesses the various marketplace and paymentapplications 120 and 122 via the web interface supported by the webserver 116. Similarly, the programmatic client 108 accesses the variousservices and functions provided by the marketplace and paymentapplications 120 and 122 via the programmatic interface provided by theAPI server 114. The programmatic client 108 may, for example, be aseller application (e.g., the TurboLister application developed by eBayInc., of San Jose, Calif.) to enable sellers to author and managelistings on the networked system 102 in an off-line manner, and toperform batch-mode communications between the programmatic client 108and the networked system 102.

FIG. 1 also illustrates a third party application 128, executing on athird party server machine 130, as having programmatic access to thenetworked system 102 via the programmatic interface provided by the APIserver 114. For example, the third party application 128 may, utilizinginformation retrieved from the networked system 102, support one or morefeatures or functions on a website hosted by the third party. The thirdparty website may, for example, provide one or more promotional,marketplace, or payment functions that are supported by the relevantapplications of the networked system 102.

FIG. 2 is a block diagram illustrating marketplace and paymentapplications 120 and 122 that, in one or more embodiments, are providedas part of application server(s) 118 in the networked system 102. Theapplications 120 and 122 may be hosted on dedicated or shared servermachines (not shown) that are communicatively coupled to enablecommunications between server machines. The applications 120 and 122themselves are communicatively coupled (e.g., via appropriateinterfaces) to each other and to various data sources, so as to allowinformation to be passed between the applications 120 and 122 or so asto allow the applications 120 and 122 to share and access common data.The applications 120 and 122 may access one or more databases 126 viathe database servers 124.

The networked system 102 may provide a number of publishing, listing,and price-setting mechanisms whereby a seller may list (or publishinformation concerning) goods or services for sale, a buyer can expressinterest in or indicate a desire to purchase such goods or services, anda price can be set for a transaction pertaining to the goods orservices. To this end, the marketplace and payment applications 120 and122 are shown to include at least one publication application 200 andone or more auction applications 202, which support auction-formatlisting and price setting mechanisms (e.g., English, Dutch, Vickrey,Chinese, Double, Reverse auctions, etc.). The various auctionapplications 202 may also provide a number of features in support ofsuch auction-format listings, such as a reserve price feature whereby aseller may specify a reserve price in connection with a listing and aproxy-bidding feature whereby a bidder may invoke automated proxybidding.

A number of fixed-price applications 204 support fixed-price listingformats (e.g., the traditional classified advertisement-type listing ora catalogue listing) and buyout-type listings. Specifically, buyout-typelistings (e.g., including the Buy-It-Now (BIN) technology developed byeBay Inc., of San Jose, Calif.) may be offered in conjunction withauction-format listings, and allow a buyer to purchase goods orservices, which are also being offered for sale via an auction, for afixed-price that is typically higher than the starting price of theauction.

Store applications 206 allow a seller to group listings within a“virtual” store, which may be branded and otherwise personalized by andfor the seller. Such a virtual store may also offer promotions,incentives, and features that are specific and personalized to arelevant seller.

Reputation applications 208 allow users who transact, utilizing thenetworked system 102, to establish, build, and maintain reputations,which may be made available and published to potential trading partners.Consider that where, for example, the networked system 102 supportsperson-to-person trading, users may otherwise have no history or otherreference information whereby the trustworthiness and credibility ofpotential trading partners may be assessed. The reputation applications208 allow a user (for example, through feedback provided by othertransaction partners) to establish a reputation within the networkedsystem 102 over time. Other potential trading partners may thenreference such a reputation for the purposes of assessing credibilityand trustworthiness.

Personalization applications 210 allow users of the networked system 102to personalize various aspects of their interactions with the networkedsystem 102. For example a user may, utilizing an appropriatepersonalization application 210, create a personalized reference page atwhich information regarding transactions to which the user is (or hasbeen) a party may be viewed. Further, a personalization application 210may enable a user to personalize listings and other aspects of theirinteractions with the networked system 102 and other parties.

The networked system 102 may support a number of marketplaces that arecustomized, for example, for specific geographic regions. A version ofthe networked system 102 may be customized for the United Kingdom,whereas another version of the networked system 102 may be customizedfor the United States. Each of these versions may operate as anindependent marketplace or may be customized (or internationalized)presentations of a common underlying marketplace. The networked system102 may accordingly include a number of internationalizationapplications 212 that customize information (and/or the presentation ofinformation by the networked system 102) according to predeterminedcriteria (e.g., geographic, demographic or marketplace criteria). Forexample, the internationalization applications 212 may be used tosupport the customization of information for a number of regionalwebsites that are operated by the networked system 102 and that areaccessible via respective web servers 116.

Navigation of the networked system 102 may be facilitated by one or morenavigation applications 214. For example, a search application (as anexample of a navigation application 214) may enable key word searches oflistings published via the networked system 102. A browse applicationmay allow users to browse various category, catalogue, or inventory datastructures according to which listings may be classified within thenetworked system 102. Various other navigation applications 214 may beprovided to supplement the search and browsing applications.

In order to make listings available via the networked system 102 asvisually informing and attractive as possible, the applications 120 and122 may include one or more imaging applications 216, which users mayutilize to upload images for inclusion within listings. An imagingapplication 216 also operates to incorporate images within viewedlistings. The imaging applications 216 may also support one or morepromotional features, such as image galleries that are presented topotential buyers. For example, sellers may pay an additional fee to havean image included within a gallery of images for promoted item listings.

Listing creation applications 218 allow sellers to conveniently authorlistings pertaining to goods or services that they wish to transact viathe networked system 102, and listing management applications 220 allowsellers to manage such listings. Specifically, where a particular sellerhas authored and/or published a large number of listings, the managementof such listings may present a challenge. The listing managementapplications 220 provide a number of features (e.g., auto-relisting,inventory level monitors, etc.) to assist the seller in managing suchlistings. One or more post-listing management applications 222 alsoassist sellers with a number of activities that typically occurpost-listing. For example, upon completion of an auction facilitated byone or more auction applications 202, a seller may wish to leavefeedback regarding a particular buyer. To this end, a post-listingmanagement application 222 may provide an interface to one or morereputation applications 208, so as to allow the seller conveniently toprovide feedback regarding multiple buyers to the reputationapplications 208.

Dispute resolution applications 224 provide mechanisms whereby disputesarising between transacting parties may be resolved. For example, thedispute resolution applications 224 may provide guided procedureswhereby the parties are guided through a number of steps in an attemptto settle a dispute. In the event that the dispute cannot be settled viathe guided procedures, the dispute may be escalated to a third partymediator or arbitrator.

A number of fraud prevention applications 226 implement fraud detectionand prevention mechanisms to reduce the occurrence of fraud within thenetworked system 102.

Messaging applications 228 are responsible for the generation anddelivery of messages to users of the networked system 102 (such as, forexample, messages advising users regarding the status of listings at thenetworked system 102 (e.g., providing “outbid” notices to bidders duringan auction process or to provide promotional and merchandisinginformation to users)). Respective messaging applications 228 mayutilize any one of a number of message delivery networks and platformsto deliver messages to users. For example, messaging applications 228may deliver electronic mail (e-mail), instant message (IM), ShortMessage Service (SMS), text, facsimile, or voice (e.g., Voice over IP(VoIP)) messages via the wired (e.g., the Internet), plain old telephoneservice (POTS), or wireless (e.g., mobile, cellular, WiFi, WiMAX)networks 104.

Merchandising applications 230 support various merchandising functionsthat are made available to sellers to enable sellers to increase salesvia the networked system 102. The merchandising applications 230 alsooperate the various merchandising features that may be invoked bysellers, and may monitor and track the success of merchandisingstrategies employed by sellers.

The networked system 102 itself, or one or more parties that transactvia the networked system 102, may operate loyalty programs that aresupported by one or more loyalty/promotions applications 232. Forexample, a buyer may earn loyalty or promotion points for eachtransaction established and/or concluded with a particular seller, andmay be offered a reward for which accumulated loyalty points can beredeemed.

The question and answer module 234 provides one or more users acommunication medium. The question and answer module 234 can be used,for example, to provide customer support to users, answer user questionsregarding products or services offered in an auction or other listing,or provide a feedback mechanism for users to review a product orservice, provide information to other users, or provide informationabout them.

The contextual information module 236 can determine a context of auser's access to a website. The contextual information module 236 cananalyze message or packet traffic from the client. The contextualinformation module 236 can receive information from the database server124 indicating item listings that a user might be interested in. Thecontextual information module 236 can rank the item listings from thedatabase server 124 based on the contextual information.

The quality score module 238 can determine a quality score of an itemlisting, such an item listing indexed in the database 126. The qualityscore module 238 can consider an item listing freshness (i.e. howrecently the item listing was put on sale or posted for sale on thenetwork), an item listing quality (i.e. a condition of the item listing,such as new, like new, good, slightly used, used, poor, etc.), an imagequality (e.g., resolution, clarity, etc. of the image), cover itemlisting quality (in the case that the item listing is a collection ofitem listings), a cover image quality (in the case that the item listingis a collection of item listings), a mean item listing quality (e.g., anaverage of the highest quality item listings of a collection of itemlistings), a mean image quality (e.g., an average of the highest qualityimages of a collection of item listings), a number of item listings inthe collections of item listings, an item listing age, an item listinglast update time/date, a number of user item listing views, a number ofuser follows, a number of user unfollows, a number of user shares, userengagement with the item listing, and/or an author of the item listing,among others.

A recommendation module 240 can receive information, such item listingrankings, a quality score, user attributes, and/or contextualinformation associated with a device a user is using to access thewebsite and recommend an item listing to a user based on the receivedinformation. The recommendation module 240 can implement a random forestclassifier or other technique in determining the item listing to berecommended.

FIG. 3 illustrates, by way of example, an embodiment of an RS 300. TheRS 300 as illustrated includes a recommendation module 240, one or moredatabases 126, one or more database servers 124, an optional contextualinformation module 236, one or more client machines 112 or 110, and/orone or more 3^(rd) party servers 130. The client machines 110 or 112 canissue a request to the network 104 (not shown in FIG. 3, see FIG. 1)that can be reformatted by the web server 116 or the API 114. Thecontextual information module 236 can analyze the request and determinecontextual information of a user's access to the network 104. Byanalyzing the request from the client machine 112 or 110 contextualinformation, such as a device type (e.g., mobile, laptop), a make and/ormodel of the device, an OS, a browser, or the like, can be determined.The contextual information can be provided directly to therecommendation module 240 by the contextual information module 236and/or the contextual information can be stored to the database(s) 126,such as through a communication the database server 124 through theconnection 314.

The selected item listing(s), selected by the recommendation module 240and to be presented to the user, can be communicated to the clientmachine 110 or 112, or the 3^(rd) party server 130, through theconnections 320, 318, and 316, respectively. The recommended itemlisting(s) (i.e. the selected item listing(s)) can be communicated tothe web server 116 or API 114, which can then communicate the selecteditem listing(s) to the network 104, which can communicate with theclient machine 110 or 112 to cause the machine to display an image ofthe selected item listing(s).

Item listings, user attributes, user contextual information, and/orpre-filtered item listing(s) can be stored and indexed in thedatabase(s) 126. The recommendation module 240 can retrieve contextualinformation from the database 126 through the connection 314, userattributes from the database 126 through the connection 312, itemlisting(s) from the database 126 through the connection 308, and canretrieve pre-filtered item listing(s) through the connection 306.

The user attributes 312 can be provided by the user, such as through theclient 110 or 112 and/or the question and answer module 234. Thecontextual information 310 can be determined by analyzing data trafficbetween the client 110 or 112 and the network 104, web server 116, orAPI server 114. For example, a MAC address may indicate a device type,browser, time or other contextual information associated with a user'saccess the network 104.

Item listing(s) can either be selected online or offline. Online meaningthat a recommendation technique to determine a recommendation for agiven user can be performed after the given user has accessed thenetwork 104, and offline meaning that the recommendation technique isperformed so that an item recommendation is pre-computed for the givenuser independent of a user's access to the network 104. To help inperforming the recommendation technique offline, pre-filtered itemlisting(s) can be stored and indexed (i.e. catalogued) in thedatabase(s) 126. The catalogued pre-filtered item listing(s) can beretrieved from the database(s) 126, such as through the connection 306.The pre-filtered item listing(s) can include item listing(s) filteredthrough one or more of the quality score module 238, the ranking module304, or the recommendation module 240. The pre-filtered item listing(s)can be filtered based on one or more of the quality score, the userattributes, and the contextual information. If the pre-filtered itemlistings are filtered based on user attributes only, then the qualityscore module 238 and the ranking module 304 can be used to filter thepre-filtered item listings based on the quality score and therecommendation module 240 can filter the pre-filtered item listingsbased on the quality score. If the pre-filtered item listings arepre-filtered based on the quality score, then the recommendation module240 can further filter the pre-filtered items based on the contextualinformation and the user attributes if the user attributes areavailable. Pre-filtering based on quality score can include using theranking module 304 and the quality score module 238 to determine aquality score and rank the item listing(s) based on the quality score.Pre-filtering based on the quality score can include comparing thequality score to a threshold and removing an item listing that is lessthan the threshold.

Data has shown that an agent tends to buy more item listings using adesktop, but engages more with a recommended item listing using a mobiledevice. If the contextual information, such as can be communicated onthe connection 314, is considered in providing a recommendation, thenthe item listings that cause more engagement can be displayed to a userwhen they are on tablet, laptop, or a mobile, while item listings that auser might be interested in buying would be presented to the user whenthey are using a desktop computer. The item listings and the collectionlisting can be tagged to indicate whether they are for user engagementor purchase, for example. Other tags for the item listings or collectionlistings can include, for example, operating system (OS), device type,user type (e.g., buyer, seller, administrator, or the like), user buyersegment, user seller segment, user demographics, time, session sourcetype, and/or is owner. The tags for the item listings or collectionlistings can indicate user attributes of users that might be interestedin engaging with or purchasing the item listing or collection of itemlistings. Thus, if contextual information of one user matches more tagsof an item listing than another item listing, the item listing thatincludes more matching tags can be ranked higher in terms of contextualmatch than the item listing that includes fewer matching tags.

For example, a user using Internet Explorer to access the web server 116may be more likely interested in Microsoft item listings than Macintoshitem listings, so Microsoft item listings can include a browser tag of“Internet Explorer”. Similarly, a user using Safari to access the webserver 116 may be more likely interested in Macintosh item listings, soMacintosh item listings can include a browser tag of “Safari”.

The recommendation module 240 can include a ranking module 304. Theranking module 304 can receive data indicating one or more collectionsof item listings 306 or one or more item listing(s) 308 that couldpotentially be recommended to a user. The data can include informationdescribing the item listings 306 or collections of item listings 308,such as a unique identifier. The data can include data depicted in Table2, among other data. The ranking module 304 can include the qualityscore module 238.

The quality score module 238 can use a relevant classifying techniquethat provides the rank for each item or collection of items. Theclassifying technique can include a random forest classifier, logisticregression, stochastic gradient descent, Gaussian naïve Bayes, Adaboost,or other technique to determine quality scores based on the receiveddata. The quality scores can be used to provide a rank of an itemlisting or a collection. For example, the item listing or collectionwith the highest relative quality score can be ranked highest (e.g.,associated with a number one) and the item listing or collection withthe lowest relative score can be ranked lowest (e.g., associated with alarger number).

The ranking module 304 can extract features from a collection/itemlisting using a feature importance technique (e.g., information gain,entropy, random forest, among others). The relative rank of each featurecan be determined by obtaining the depth of the random forest classifiergenerated while using each attribute as a decision node. Less depth canimply more importance.

The recommendation module 240 can consider the quality score and/or therank of the item listings in providing a recommendation of the itemlistings. The recommendation module 240 can rank potential item listingsbased on contextual information 310 (e.g., one or more of the itemlistings listed in Table 1) or user attributes 312 before providing therecommendation. The quality score from the quality score module 238, thecontextual information 310, and the user attributes 312 can be used tofilter (e.g., rank) the item listings in a variety of orders, such ascan produce different results. Three ways that are discussed hereininclude: (i) contextual pre-filtering (FIG. 4); (ii) contextualpost-filtering (FIG. 5); and (iii) contextual modeling (FIG. 6). Therecommendation module 240 can recommend an item listing/collection usingany of the techniques discussed herein, including contextualpre-filtering (technique 400), contextual post-filtering (technique500), and contextual modeling (technique 600).

The three techniques and variations thereof are discussed and acomparative performance analysis between contextual pre-filtering,contextual post-filtering, and contextual modeling are presented inFIGS. 5, 6, and 7. These techniques are analyzed and some of theirperformance characteristics are presented. An analysis of prior CARSs ispresented with a discussion of some of the drawbacks that can beovercome using one or more of the embodiments discussed herein

Conventional CARSs can suffer from data sparseness issues. Datasparseness can be caused when there is little or no information about auser. Some RSs are configured to provide a recommendation of an itemlisting that is popular when there is little or no information about auser. Thus, some RSs tend to become popularity based, such as can be dueto data sparseness. Such systems tend to give bad recommendations whenuser attributes are not available. Also, such systems can lack indiversity. To help in solving one or more of these issues, in anembodiment, a versatile ranking mechanism (e.g., using the RS 300) isprovided that calculates a quality score of various item listings orcollections of item listings based on item listing level, collectionlevel, and/or historical performance features of the item listings orcollections. Using this ranking, all the item listings are ratedindependently of the user ratings, contextual information, and userattributes, hence there is no sparseness in information for itemlistings. Contextual information (such as device type: mobile or desktopetc.) and/or a user's attributes can be used to pre-filter the itemlistings, post-filter the item listings, or be combined with the qualityscores of the collections for better recommendation personalization.

Using the quality score, new item listings or collection of itemlistings can be recommended to the users with improved accuracy.Filtering (e.g., ranking) potential item listings based on qualityscores of item listings or collections can lead to betterrecommendations. Unlike other approaches, one or more of the approachesdiscussed herein does not use a popularity base in providing arecommendation. The experimental results discussed show a performanceboost and better personalization.

Another challenge with existing RSs is that even if item listings aresold out they may still be recommended. Item listings can sell outquickly. The proposed ranking mechanism(s) can consider item listingage, item listing last update time, and other such features into accountto help in preventing recommending an item listing that has sold out.These features of the item listings can be considered in determining thequality score of an item listing. Item listings which are sold out andpopular can be ranked lower. An approach discussed herein can refrainfrom recommending an item listing which is sold out and popular.

An advantage of an approach discussed herein is that it can be built ontop of an already existing RS, irrespective of its type. In some RSs, itmay be difficult to recommend an item listing to a user when userattributes are unknown. Discussed herein are a variety of mechanisms toprovide a recommendation based on contextual information and qualityscores of collections so as to provide a recommendation independent ofuser attributes. Contextual information is available most of the time sorecommendations are unlikely to be based on the quality score alone.

In this disclosure, recommendation techniques are discussed in thecontext of using a random forest classifier technique. While a randomforest classifier technique is used as an example technique, othertechniques can be used, such as those shown in FIGS. 7, 8, and 9 anddiscussed herein.

One or more contextual features can be considered in an RS, in one ormore embodiments. Contextual information includes device type, operatingsystem (OS), browser, etc. Since contextual information is usuallyavailable, imputation may be avoided (i.e. using imputation, where thereis data loss or data is not available, data is extrapolated or otherwisegenerated to fill in an information gap), and over-fitting of the modelis avoided. Imputation and over-fitting are some common issues withconventional RSs. Table 1 shows a list of some contextual features thatcan be considered in providing a recommendation (i.e. a recommendationfor an item listing, such as a product for purchase, a video forviewing, a website for viewing, a story for reading, etc.).

TABLE 1 Example Contextual Features Contextual Features Visitor TypeVisitor Buyer Segment Visitor Seller Segment Demographics Time OS TypeDevice Type (Desktop/Mobile/Tablet) Session Source Type isOwner

Using a random forest classifier technique in an RS (e.g., apersonalized and/or context aware RS, or other RS) to rank potentialrecommendation item listings can help alleviate one or more of thedrawbacks of the conventional RSs, such as the RSs and/or drawbackspreviously discussed. The ranking mechanism can be integrated with anexisting RS to provide a better recommendation experience to a user. Aquality score for an item listing or collections of item listings thatmight be recommended can be calculated using this ranking mechanism andthe item listings/collections can be filtered based on the qualityscore.

An RS as discussed herein can include one or more advantagesincluding: 1) incorporating contextual information can cause betteruser-engagement. For example, different personalization models fordifferent device-types being used to access a website can help provide abetter user experience; 2) new item listings/collections, and thosewithout historical performance can be recommended more accurately; 3)sparseness issues can be reduced by providing a new user with arecommendation based on a quality score of the item listing orcollection and/or contextual information; 4) the recommendation can be agood recommendation even if a user's past preference/purchases is notavailable. In such cases, contextual information can be used inproviding the recommendation since contextual information is almostalways available; 5) using a random forest classifier for calculatingquality scores and leveraging item listing/collections characteristics,users preferences, or contextual information for recommendations can becomputationally cheaper than existing context aware techniques; 6) noadditional techniques, such as dimensionality reduction (O(n²)), ortechniques to deal with sparseness are required; 7) random forest isO(n*m*log(n)), where n is the number of samples and m is the number ofdecision trees used. This is computationally much cheaper thansimilarity matching used in CF, which is (O(n²)); 8) there is lessdependency on user ratings of item listings. Most of the existingrecommendation systems (e.g., Yelp, Netflix) use user ratings, which cancause dependency issues and lead to sparseness; 9) an inherent rankingsystem to obtain a quality score can be more objective than user (human)rating of item listings/collections; 10) there is no need to deal withregularization to avoid over-fitting; or 11) sold out item listings arenot recommended, even if they are popular.

The ranking mechanism (e.g., a quality score as determined by thequality score module 238) for ranking an item listing or a collectioncan consider item listing characteristics, collections features,historical performance of an item listing, type of author (those whomade the collection/item listing), among others. This mechanism can rankold collections and item listings, and can rank new collections and itemlistings as well. The ranking mechanism (as implemented by the rankingmodule 304) can be integrated along with contextual information and/or apersonalized RS to recommend an item listing or a collection of higherquality (e.g., an item listing or a collection with a higher qualityscore) in the absence of other user or context information.

The random forest classifier can also be used for generating apersonalized recommendation. If a default personalized recommendationmodel that incorporates users-item listings ratings or users pastpreferences, etc. for recommendations already exists, then the samemodel can be integrated with contextual information using a technique orsystem discussed herein. So, the proposed methods and systems are robustin that they can be added to an existing RS or can be used on their own.

Item listings can be ranked based on a user rating. Sometimes auser-rating is not available from a specific user. Not every websiteprovides a rating system for an item listing or collection of itemlistings. The eBay Collections website (http://www.ebay.com/cln) fromeBay, Inc. of San Jose, Calif., is such a website. In such cases,instead of using the user rating for an item listing or collection,these item listings or collections can be ranked on popularity(followers, shares, or the like, among others), past purchases, or acombination thereof. A disadvantage to such a ranking is that popularcollections become more popular, and even if the item listing is soldout it can still be recommended, as previously discussed. In addition,such ranking generally does not provide recommendations to newcollections. To help alleviate these issues, a last-update time or acollection-age of the collection can be considered by a random forestclassifier in ranking the item listing or collection of item listings.However, this might lead to recommendations of collections of badquality, because a collection that was updated recently, does not ensurethe collection is of a high quality. Care may be taken not to put toomuch weight on a last update time, or a user rating or popularity of anitem listing or collection.

The ranking, as provided by the random forest classifier can be based onitem listing level attributes (e.g., item listing freshness, itemlisting quality, image quality, and item listing past performance, amongothers), collection level attributes (e.g., number of item listings,collection age, collection past performance, collection last updatetime, among others), author level attributes (e.g., type of author, suchas celebrities, artists, or professionals, among others) and/or historicperformance of the item listing or collection of item listings (e.g.,followers, shares, or registrations, among others). Note that someauthors are curators (e.g., celebrities, artists, or professionals) andtheir collections are of very high quality. See Table 2 for adescription of example parameters that can be considered in a qualityscore.

TABLE 2 Some parameters that can be used in determining a quality scoreCategory Features 1. Item listings Item listing Freshness Item listingQuality Image Quality Cover Item listing Quality Cover Image Quality Top5 Mean Item listing Quality Top 5 Mean Image Quality 2. CollectionsNumber of Item listings Collection's Age Collection's Last Update 3.Historical Performance View Item listings (3, 7, 15, 30, 45, 60 days)View Item listing Layer Counts Follow Counts Unfollow Counts ShareCounts User Engagement (BBOWAA*) 4. Author Author Type Author BuyerSegment Author Seller Segment isCurator (celebrity/paid professionalauthor)

Given a collection or item listing, a quality score can be defined asthe probability of the item listing or collection havinguser-engagement. Multiple metrics can be used to measureuser-engagement. Two metrics are considered here: 1) VI (view itemlisting: user views item listing in that collection), BBOWAA (Bids,Bins, Offer, Watch counts, Add to cart, or Ask seller a question). Usingthese metrics a machine-learning based model can determine whetheruser-engagement will happen or not. Determining the quality score can beconsidered as a binary classification problem in one or moreembodiments, such as where class “1” corresponds to engagement and class“0” corresponds to no-engagement. However, this disclosure is notlimited to such a classification, a ternary classification or otherclassification can be used. For example, a “00” can correspond to noengagement, “01” can correspond to a quick engagement (e.g., a click andview time of less than ten seconds), a “10” can correspond to a regularamount of engagement (e.g., a view time of greater than ten seconds),and a “11” can correspond to a greater than normal amount of engagement(e.g., VI and BBOWAA). The quality score can be calculated offline sothat it is ready for use when a user accesses a website.

In the binary classification scheme, the probability with which class“1” is predicted is defined as the quality score. Because the data hasrandomness (e.g., due to user's behavior), it can be helpful to removefalse negatives, false positives, or try to control the variance. Anensemble technique can be used for determining the quality score. Usinga random forest classifier for ranking an item listing or collection, apercentage of classifiers that predicted a class “1” can be divided bythe total number of all sub-classifiers in the random forest classifierto determine the quality score. So, if 60% of all the sub-classifierspredicted class “1” then the quality score of that collection can be0.6, for example.

A quality score can be generated for a new collection or item listing,such as by using item listing or collection level attributes. With time,historic performance parameters can be incorporated into the qualityscore for an item listing or collection. Models can be updatedperiodically (e.g., daily, hourly, weekly, monthly, etc.), randomly, oron request. Choosing to update often enough, but not too often, canprovide enough time for the item listings in most collections to be soldor otherwise changed, and can allow new collections to build somehistorical information.

To rank an item listing, some of the best features of the item listingcan be extracted, by the ranking module 304, using a feature importancetechnique (e.g., information gain, entropy, random forest, amongothers). The relative rank of each feature can be determined byobtaining the depth of the random forest tree generated while using eachattribute as a decision node. Less depth can imply more importance.Table 2 summarizes some of the features that were considered at thecollections level for simulations discussed herein.

A quality of a recommendation can be improved when compared to aconventional RS or a prior personalized RS using one or more of thetechniques discussed herein. Also discussed herein is how contextualinformation can be used to help improve personalized recommendations. Inthe context of a recommendation, a high quality recommendation is onethat a user engages with. An improvement in an RS includes a userengaging with a recommendation with a higher probability, the RSproviding the recommendation using fewer computer resources whilemaintaining or improving the probability a user takes therecommendation, and/or an RS providing the recommendation in less timewhile maintaining or improving the probability a user takes therecommendation, among others.

FIG. 4 illustrates, by way of example, a flow diagram of an embodimentof a contextual pre-filtering technique 400. In contextualpre-filtering, based on user's context, item listings are alreadypre-filtered out offline. The item listings can then be further filtered(e.g., ranked) online based on the user attributes and/or the qualityscores of the item listings. Those item listings which are relevant tothe context in the current session are reflected and others arepre-filtered out. For example, users tend to engage more on tablets,laptops, and mobile phones while they tend to purchase more using adesktop. Also, users are generally more active on desktops during theday and morning time and engage through tablets and mobile in eveningand night. The item listings can be filtered (e.g., using the database124 and the database server 126) based on the context of the useraccess. After the item listings are filtered based on the contextualinformation then the item listings can be filtered again based on: (i) auser's attributes at operation 406 (e.g., personalized search results,purchase history, browsing history, demographics, or other attributes)and/or (ii) based on quality scores at operation 404. If the userattributes are not available (which can be determined at operation 408or operation 410), then recommendations can be solely based on qualityscores 404 and pre-contextual filtering 402. If user attributes areavailable for the user, the item listings and collections can befiltered based on the user attributes at 406 and then filtered based onthe quality score 404.

This approach works well online, that is, in the current session.Pre-filtering tends to lead to local models instead of a single globalmodel. Every context (and related contexts) might end up being onerelatively small local model. A drawback of this approach is sparseness,since, after pre-filtering is performed, there may be no training data(e.g., item listings) available for that context. A bad recommendationcould be provided in such a situation. Pre-filtering also does not scalewell with many contextual variables. One or more of the item listingscan be recommended from the item listings or collections remaining afterall the filtering.

FIG. 5 illustrates, by way of example, a flow diagram of an embodimentof a contextual post-filtering technique 500. Unlike contextualpre-filtering, as shown in FIG. 4, item listings are filtered based oncontextual information in the post-process (i.e. at operation 310) incontextual post-filtering technique 500. Whether user attribute data isavailable can be determined at 240 and 304. The item listing(s) orcollection(s) can be filtered based on the quality score at 306 or 308.If user data is available the item listing(s) or collection(s) can befiltered based on user attributes and the quality score at 308. Thefiltering of the item listings based on the quality score and/or theattribute data can be done offline or online, such as by the database124 or the database server 126 or the ranking module 304 and/or thequality score module 238. If no user data is available then the userattributes are not available and the item listing(s) are filtered basedon quality scores at operation 306. After filtering based on qualityscore and/or user attributes, the item listing(s) or collection(s) canbe filtered based on contextual information 310, such as online. One ormore item listings or collections can be recommended based on the itemlistings or collections remaining after all the filtering.

In many cases post-filtering out-performs pre-filtering in providingmore relevant results and also in terms of execution time.Post-filtering also integrates well with any existing RS. This allows analready existing RS to be used offline for personalization. If user'sattributes are not available or if the user is new, then the initialfiltering can be done using only quality score at operation 306.Generally, a number of contextual features are limited and much lessthan those incorporated in a personalized recommendation, andpersonalized recommendations are generally pre-calculated offline, whichgets updated for every new session.

FIG. 6 illustrates, by way of example, a flow diagram of an embodimentof a contextual modeling technique 600. In contextual modeling,contextual features are integrated with those used for personalization(e.g., user attributes). The models are trained (e.g., the itemlisting(s) or collection(s) are filtered) with all the features:contextual and non-contextual. Contextual modeling is computationallyexpensive when compared to contextual post-filtering and contextualpre-filtering.

At operations 402 and 404, it can be determined whether user attributedata is available. If user attribute data is not available, itemlistings can be filtered based on only the contextual information atoperation 406. The item listings can further be filtered based on aquality score at operation 410. At operation 408, item listings can befiltered based on user attributes and contextual information. Thefiltered item listings after performing operations 408 or 406 can befiltered based on quality scores at operation 410. One or more itemlistings or collections remaining after all the filtering can berecommended to the user.

As used herein, “filter” means to rank and/or remove, such as by using arandom forest classifier or other technique and/or comparing the rank toa threshold to determine if the item listing is to be removed. In usinga random forest classifier, changing the definition of one or more nodesgenerally changes the forest, generally leading to different numbers ofsub-classifiers including an engagement prediction from the user. Thus,the definition of the nodes can change the predicted outcome and thefinal recommendation. The result of a filter process can include a listor set of one or more item listings or collections of item listings. Theresult can include a relative rank number that can indicate an itemlisting or collection is generally more or less likely to cause a userto engage with the item listing or collection.

Contextual information can change often. When modeling, it should beconsidered that a recommendation adheres to the context in action. Sincea user might be involved in different kinds of context in differentsessions, it is difficult to train the model offline according to allpossible contexts for every session in real-time (online). Therefore, incontextual modeling some heuristics are generally applied. A model isgenerally trained with those contexts that are most common for thatuser. Thus, a user might use a desktop and a tablet, but while trainingthat device type can be chosen which is most frequent for that user.

Incorporating every combination of context for a user becomescomputationally expensive and can cause over-fitting. Contextualmodeling can work well with respect to every session, when the model istrained offline. The contextual models are generally static models,confined to a session. What is recommended in a next session can bepre-defined or determined after a session. The model can be refreshedbefore the next session to determine the next recommendation.

The historical performance of item listings/collections for differentperiods was considered. Historical performance of collections in pastthree days, seven days, fifteen days, thirty days, forty-five days, andsixty days were considered. Models with seven days historicalperformance gave better accuracy than others. The best duration torefresh the models being tested was about seven days. Seven days gaveenough time for new collections/item listings to mature, to know theperformance of old collections/item listings, and allow collections/itemlistings to sell out.

Various classification techniques were used to obtain whether engagementwould happen or not. A binary classification technique was used.Although, most of the classification techniques that were chosen gavegood results, a random forest classification technique out-performed allother approaches. A probability of a user engaging with an item listingor collection was determined from the models; this probability is calledthe quality score of that item listing or collection. Tables 3, 4, and 5(See FIGS. 5, 6, and 7, respectively) depict the performance ofdifferent models. The analysis was performed at collections level and atsession level. A comparison of the performance of three differentparadigms of context aware recommendations is presented.

The analysis was conducted at session level and collection (of itemlistings) level. Session level information was collected in response toa user visiting a collection during a session. A session is a uservisiting a website and visiting an item listing in a collection.

Collection level information is the aggregated data (such as performance(e.g., sales, user-engagement, or the like), item listing quality, imagequality, etc.) for all the visits that happened in a collection. So, ina given time period there could be multiple visits in the samecollection from the same user. Collection and item listing featurevalues are dynamic and can change frequently. Analysis was performed onaround twelve million visits, five hundred thousand collections, andthirteen million item listings.

Collection level data can be used by the ranking module 304 to obtain aquality score of an item listing or a collection. The collection leveldata can also be integrated with user's attributes to obtain apersonalized recommendation. The techniques herein determine probabilityof a user engaging with an item listing assuming an item listing isshown to a user. If the probability is greater than a threshold, (e.g.,0.5 or other threshold), then the collection can be classified as class“1”. At session level, a predicted order of a user visiting a pluralityof the collections (e.g., all the collections) was calculated, givensome contextual features and users past preferences (if present).Integrating quality score of collections with the personalization andcontextual models yielded better results than other techniques.

Since e-Commerce business is dependent on recommendations and suchorganizations tend to develop their own RS, it is very difficult toreplace them. In such cases, the ranking system, which can be used toobtain the quality scores can be used in addition to the alreadyexisting recommendation score. Quality scores, contextual data, andalready existing personalized recommendations (if present) can deliver amuch better experience for a user as compared to CB, CF, or acombination thereof.

FIG. 7 illustrates, by way of example, a table 700 depicting simulationresults of a contextual modeling recommendation vs. context-lessrecommendation at the collections level. FIG. 7 shows a performanceboost, when incorporating context (i.e. contextual modeling) in themodels at collections level. There is an improvement of up to 19% in thearea under ROC curve, also the RMSE is reduced by up to about 37% usingthe random forest classifier technique. The added computational cost ofthe random forest classifier is relatively small.

FIG. 8 illustrates, by way of example, a table 800 depicting simulationresults of a contextual modeling recommendation vs. context-lessrecommendation at the session level. Less improvement was observed atsession level when compared to recommendations at collections level.There was up to about 12% improvement over using a stochastic gradientdescent method and about a 10% improvement using a random forestclassifier in the area under ROC. A decrease of 7% RMSE is obtained withthese models.

It is relatively hard to predict the engagement of users at sessionlevel when compared to collections level. If a collection is good, thenit tends to have more engagement, however an active user might notengage with all the collections. Adding contextual information improvesthe performance.

FIG. 9 illustrates, by way of example, a table 900 depicting simulationresults of a contextual pre-filtering and context post-filtering. In thestudy pre-filtering gave similar performance results as contextualmodeling while post-filtering gave better performance and in less time.

FIG. 10 illustrates, by way of example, a flow diagram of an embodimentof a method 1000. The method 1000 can be implemented using the system300, the contextual information module 236, and/or one or more othermodules, databases, or database servers discussed herein. The method1000 as illustrated includes: determining, using the quality scoremodule 238, a quality score for a plurality of item listings orcollections of item listings, at operation 1002; and recommending, usinga recommendation module 240, an item listing or collection of itemlistings based on the quality score, at operation 1004. The qualityscore can be determined independent of a user's attributes andindependent of contextual information. The contextual information cancorrespond to details of the user's access to a website. The operationat 1004 can include recommending an item listing or collection of itemlistings to a user based on one or more of the quality score, userattributes, and the contextual information.

The operation at 1002 can include determining the quality score using arandom forest classifier, a logistic regression, a stochastic gradientdescent, a Gaussian naïve Bayes, an Adaboost, another technique, or acombination thereof. The method 1000 can include comparing a qualityscore to a threshold. In one or more embodiments, the item listing orcollection associated with the quality score can be recommended only ifthe quality score is greater than the threshold. The contextualinformation can include at least one of a type of device the user isusing to access the website, a user type, a user buyer segment, a userseller segment, a user demographic, a time of access, an operatingsystem type of the device of the user, a session source type, andwhether the user is an owner of an item listing for sale on the website,among others.

The method 1000 can include contextual pre-filtering, contextualpost-filtering, or contextual modeling the item listings or collectionof item listings before recommending the item listing or collection ofitem listings. Contextual pre-filtering can include pre-filtering theitem listings or collections based on the contextual information,determining whether user attributes are available for the user.Contextual pre-filtering can include, in response to determining userattributes are available, further filtering the pre-filtered itemlistings or collections based on the user attributes and filtering thefurther filtered item listings or collections based on the determinedquality score for the recommended item listing or collection of itemlistings to create a first filtered set and wherein recommending an itemlisting or collection of item listings to the user includes choosing anitem listing from the first filtered set. Contextual pre-filtering caninclude, in response to determining the user attributes are notavailable, further filtering the pre-filtered item listings orcollection based on the quality scores to create a second filtered setand wherein recommending an item listing or collection of item listingsto the user includes choosing an item listing from the second filteredset.

Contextual post-filtering can include determining if user attributes areknown. Contextual post-filtering can include, in response to determininguser attributes are not known, providing the recommendation using thequality score filtered by the contextual information. Contextualpost-filtering can include, in response to determining the userattributes are known, recommending the item listing or collection ofitem listings to a user chosen based on a quality score modifiedfiltered by the user attributes and then filtered by the contextualinformation.

Contextual modeling can include determining if user attributes areknown. Contextual modeling can include, in response to determining userattributes are not known, providing the recommendation using thecontextual information filtered by the quality score. Contextualmodeling can include, in response to determining the user attributes areknown, recommending the item listing or collection of item listings to auser chosen based on user attributes and contextual information and thenfiltered by the quality score.

A random forest classifier technique is an ensemble machine learningtechnique for classification that includes the construction of aplurality of decision trees at training time. Training for a randomforest generally uses bootstrap aggregating to a tree learner. Giventraining data, X=x₁, . . . , x_(n) with response Y=y₁, . . . , y_(n),the training selects random samples of the training data, X and fits thesample to a regression tree on the selected random samples. A predictionfor a new sample can be made by averaging the predictions from theindividual regression trees on the new sample or by taking a majorityvote on the decision trees (e.g., a majority of “1” or a majority of“0”). A random forest classifier uses a modified tree learning techniquethat selects, at each node (“split”) a random subset of the branch arechosen. For each branch a random value can be selected in the node'srange, the best random value can be chosen as the node.

In the context of providing an item listing or collectionrecommendation, each item listing or collection can be considered avariable and the “importance” (e.g., quality score or relative rank) ofthe variable can be determined using the random classifier technique.First, a random forest can be fit to the data. Error in fitting can berecorded. To measure the importance of a variable, the values of thevariable are permuted among the training data and the error is computedagain the altered data set. The importance of the variable is computedby averaging the difference in error before and after permuting thevariable. The importance can be normalized by the standard deviation ofthe differences. The higher the importance the number, the higher therank of the variable.

A logistic regression technique includes a binomial or multinomialregression. Logistic regression can be used to predict the probabilitythat an outcome is true based on a value of a variable. In logisticregression it is assumed that there are observed data points, N. Eachdata point of N consists of features and output Y. The goal of thetechnique is to determine a relationship between the features and theoutcome. For a latent (unknown) variable, a distribution of the latentvariable can be determined based on a predictor function and a randomerror variable that is distributed according to the logistic regressionmodel.

Logistic regression is a conditional distribution of the output Y, giventhe features, X. The logistic regression can include determining thisprobability based on two coefficients. One coefficient is determined bysetting a linear regression predictor to zero and the other is aregression coefficient that regulates how fast a probability changeswith a changing feature. The regression coefficient can be selected asthe value that minimizes the sum of squares.

A stochastic gradient descent technique is an optimization method forminimizing an objective function that is written as a sum ofdifferentiable functions. In the case of providing a recommendation

A Gaussian naïve Bayes technique includes a probabilistic classifierwith independence assumptions between features. That is one featurecontributes independently to a probability that the feature is a memberof a class, regardless of the presence or absence of other features.Features can be classified into categories, such as relevant to the useror not relevant to the user, for example. Training data can be used todetermine a likelihood of influencing a user by providing arecommendation. Training data can be updated periodically to help inimproving the recommendation technique.

An Adaboost technique is a method of training a boosted classifier. Aboosted classifier is a weak learner that returns a real value where thesign of the real value indicates the predicted class and the magnitudeof the weak learner gives the confidence in that classification.

In general these techniques are linear regression techniques thatattempt to fit a set of features to a user profile. The recommendationthat minimizes an error between the features and the profile can bechosen and presented to the user. The fitting can consider qualityscore, contextual features, and/or attributes, in a variety of differentorders.

Example Mobile Device

FIG. 11 is a block diagram illustrating a mobile device 1100, accordingto an example embodiment. The mobile device 1100 can include the clientmachine 110 and//or 112. The mobile device 1100 may include a processor1102. The processor 1102 may be any of a variety of different types ofcommercially available processors 1102 suitable for mobile devices 1100(for example, an XScale architecture microprocessor, a microprocessorwithout interlocked pipeline stages (MIPS) architecture processor, oranother type of processor 1102). A memory 1104, such as a random accessmemory (RAM), a flash memory, or other type of memory, is typicallyaccessible to the processor 1102. The memory 1104 may be adapted tostore an operating system (OS) 1106, as well as application programs1108, such as a mobile location enabled application that may provideLBSs to a user. The processor 1102 may be coupled, either directly orvia appropriate intermediary hardware, to a display 11010and to one ormore input/output (I/O) devices 1112, such as a keypad, a touch panelsensor, a microphone, and the like. Similarly, in some embodiments, theprocessor 1102 may be coupled to a transceiver 1114 that interfaces withan antenna 1116. The transceiver 1114 may be configured to both transmitand receive cellular network signals, wireless data signals, or othertypes of signals via the antenna 1116, depending on the nature of themobile device 1100. Further, in some configurations, a GPS receiver 1118may also make use of the antenna 1116 to receive GPS signals.

Modules, Components and Logic

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied (1) on a non-transitorymachine-readable medium or (2) in a transmission signal) orhardware-implemented modules. A hardware-implemented module is atangible unit capable of performing certain operations and may beconfigured or arranged in a certain manner. In one or more embodiments,one or more computer systems (e.g., a standalone, client or servercomputer system) or one or more processors 1102 may be configured bysoftware (e.g., an application or application portion) as ahardware-implemented module that operates to perform certain operationsas described herein.

In various embodiments, a hardware-implemented module may be implementedmechanically or electronically. For example, a hardware-implementedmodule may comprise dedicated circuitry or logic that is permanentlyconfigured (e.g., as a special-purpose processor, such as a fieldprogrammable gate array (FPGA) or an application-specific integratedcircuit (ASIC)) to perform certain operations. A hardware-implementedmodule may also comprise programmable logic or circuitry (e.g., asencompassed within a general-purpose processor 1102 or otherprogrammable processor 1102) that is temporarily configured by softwareto perform certain operations. It will be appreciated that the decisionto implement a hardware-implemented module mechanically, in dedicatedand permanently configured circuitry, or in temporarily configuredcircuitry (e.g., configured by software) may be driven by cost and timeconsiderations.

Accordingly, the term “hardware-implemented module” should be understoodto encompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired) or temporarily ortransitorily configured (e.g., programmed) to operate in a certainmanner and/or to perform certain operations described herein.Considering embodiments in which hardware-implemented modules aretemporarily configured (e.g., programmed), each of thehardware-implemented modules need not be configured or instantiated atany one instance in time. For example, where the hardware-implementedmodules comprise a general-purpose processor 1102 configured usingsoftware, the general-purpose processor 1102 may be configured asrespective different hardware-implemented modules at different times.Software may accordingly configure a processor 1102, for example, toconstitute a particular hardware-implemented module at one instance oftime and to constitute a different hardware-implemented module at adifferent instance of time.

Hardware-implemented modules can provide information to, and receiveinformation from, other hardware-implemented modules. Accordingly, thedescribed hardware-implemented modules may be regarded as beingcommunicatively coupled. Where multiple of such hardware-implementedmodules exist contemporaneously, communications may be achieved throughsignal transmission (e.g., over appropriate circuits and buses thatconnect the hardware-implemented modules). In embodiments in whichmultiple hardware-implemented modules are configured or instantiated atdifferent times, communications between such hardware-implementedmodules may be achieved, for example, through the storage and retrievalof information in memory structures to which the multiplehardware-implemented modules have access. For example, onehardware-implemented module may perform an operation, and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware-implemented module may then,at a later time, access the memory device to retrieve and process thestored output. Hardware-implemented modules may also initiatecommunications with input or output devices, and can operate on aresource (e.g., a collection of information).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors 1102 that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors 1102 may constitute processor-implementedmodules that operate to perform one or more operations or functions. Themodules referred to herein may, in some embodiments, compriseprocessor-implemented modules.

Similarly, the methods described herein may be at least partiallyprocessor-implemented. For example, at least some of the operations of amethod may be performed by one or processors 1102 orprocessor-implemented modules. The performance of certain of theoperations may be distributed among the one or more processors 1102, notonly residing within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor 1102 or processors1102 may be located in a single location (e.g., within a homeenvironment, an office environment or as a server farm), while in otherembodiments the processors 1102 may be distributed across a number oflocations.

The one or more processors 1102 may also operate to support performanceof the relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), these operations being accessible via anetwork (e.g., the Internet) and via one or more appropriate interfaces(e.g., APIs).

Electronic Apparatus and System

One or more embodiments may be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or incombinations of them. Example embodiments may be implemented using acomputer program product (e.g., a computer program tangibly embodied inan information carrier, such as in a machine-readable medium forexecution by, or to control the operation of a data processingapparatus, such as a programmable processor 1102, a computer, ormultiple computers).

A computer program can be written in any form of programming language,including compiled or interpreted languages, and it can be deployed inany form, including as a stand-alone program or as a module, subroutine,or other unit suitable for use in a computing environment. A computerprogram can be deployed to be executed on one computer or on multiplecomputers at one site or distributed across multiple sites andinterconnected by a communication network.

In one or more embodiments, operations may be performed by one or moreprogrammable processors 1102 executing a computer program to performfunctions by operating on input data and generating output. Methodoperations can also be performed by, and apparatus of exampleembodiments may be implemented as, special purpose logic circuitry(e.g., a field programmable gate array (FPGA) or an application-specificintegrated circuit (ASIC)).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. Inembodiments deploying a programmable computing system, it will beappreciated that that both hardware and software architectures requireconsideration. Specifically, it will be appreciated that the choice ofwhether to implement certain functionality in permanently configuredhardware (e.g., an ASIC), in temporarily configured hardware (e.g., acombination of software and a programmable processor 1102), or acombination of permanently and temporarily configured hardware may be adesign choice. Below are set out hardware (e.g., machine) and softwarearchitectures that may be deployed, in various example embodiments.

FIG. 12 is a block diagram illustrating an example computer systemmachine 1200 upon which any one or more of the techniques hereindiscussed can be run. In one or more embodiments, the recommendationmodule 240, contextual information module 236, quality score module 238,and/or the ranking module 304 can include one or more item listings ofcomputer system 1200. Computer system 1200 can be embodied as acomputing device, providing operations of the recommendation module 240,contextual information module 236, quality score module 238, and/or theranking module 304 or any other processing or computing platform orcomponent described or referred to herein. In alternative embodiments,the machine operates as a standalone device or can be connected (e.g.,networked) to other machines. In a networked deployment, the machine canoperate in the capacity of either a server or a client machine inserver-client network environments, or it can act as a peer machine inpeer-to-peer (or distributed) network environments. The computer systemmachine can be a personal computer (PC), such as a PC that can beportable (e.g., a notebook or a netbook) or a PC that is notconveniently portable (e.g., a desktop PC), a tablet, a set-top box(STB), a gaming console, a Personal Digital Assistant (PDA), a mobiletelephone or Smartphone, a web appliance, a network router, switch orbridge, or any machine capable of executing instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein. Implementingtechniques using computer processors and other logic can lead toautomated camera condition change detection (e.g., that does not includehuman interference).

Example computer system 1200 can include a processor 1202 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU) orboth), a main memory 1204 and a static memory 1206, which communicatewith each other via an interconnect 1208 (e.g., a link, a bus, etc.).The computer system 1200 can further include a video display unit 1210,an alphanumeric input device 1212 (e.g., a keyboard), and a userinterface (UI) navigation device 1214 (e.g., a mouse). In oneembodiment, the video display unit 1210, input device 1212 and UInavigation device 1214 are a touch screen display. The computer system1200 can additionally include a storage device 1216 (e.g., a driveunit), a signal generation device 1218 (e.g., a speaker), an outputcontroller 1228, or a network interface device 1220 (which can includeor operably communicate with one or more antennas 1230, transceivers, orother wireless communications hardware), or one or more sensors 1221,such as a GPS sensor, compass, location sensor, accelerometer, or othersensor.

The storage device 1216 includes a machine-readable medium 1222 on whichis stored one or more sets of data structures and instructions 1224(e.g., software) embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 1224 canalso reside, completely or at least partially, within the main memory1204, static memory 1206, and/or within the processor 1202 duringexecution thereof by the computer system 1200, with the main memory1204, static memory 1206, or the processor 1202 also constitutingmachine-readable media. The processor 1202 configured to perform anoperation can include configuring instructions of a memory or othermachine-readable media coupled to the processor, which when executed bythe processor, cause the processor 1202 to perform the operation.

While the machine-readable medium 1222 is illustrated in an exampleembodiment to be a single medium, the term “machine-readable medium” caninclude a single medium or multiple media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storethe one or more instructions 1224. The term “machine-readable medium”shall also be taken to include any tangible medium that is capable ofstoring, encoding or carrying instructions for execution by the machineand that cause the machine to perform any one or more of themethodologies of the present disclosure or that is capable of storing,encoding or carrying data structures utilized by or associated with suchinstructions. The term “machine-readable medium” shall accordingly betaken to include, but not be limited to, solid-state memories, opticalmedia, and magnetic media. Specific examples of machine-readable mediainclude non-volatile memory, including, by way of example, semiconductormemory devices (e.g., Electrically Programmable Read-Only Memory(EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM))and flash memory devices; magnetic disks such as internal hard disks andremovable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

The instructions 1224 can further be transmitted or received over acommunications network 1226 using a transmission medium via the networkinterface device 1220 utilizing any one of a number of well-knowntransfer protocols (e.g., HTTP). Examples of communication networksinclude a local area network (LAN), wide area network (WAN), theInternet, mobile telephone networks, Plain Old Telephone (POTS)networks, and wireless data networks (e.g., Wi-Fi, 3G, and 4G LTE/LTE-Aor WiMAX networks). The term “transmission medium” shall be taken toinclude any intangible medium that is capable of storing, encoding, orcarrying instructions for execution by the machine, and includes digitalor analog communications signals or other intangible medium tofacilitate communication of such software.

Although an embodiment has been described with reference to specificembodiments, it will be evident that various modifications and changesmay be made to these embodiments without departing from the broaderspirit and scope of the present disclosure. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense. The accompanying drawings that form a parthereof show, by way of illustration, and not of limitation, specificembodiments in which the subject matter may be practiced. Theembodiments illustrated are described in sufficient detail to enablethose skilled in the art to practice the teachings disclosed herein.Other embodiments may be utilized and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. This Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

Although specific embodiments have been illustrated and describedherein, it should be appreciated that any arrangement to achieve thesame purpose may be substituted for the specific embodiments shown. Thisdisclosure is intended to cover any and all adaptations or variations ofvarious embodiments. Combinations of the above embodiments, and otherembodiments not specifically described herein, will be apparent to thoseof skill in the art upon reviewing the above description.

The Abstract of the Disclosure is provided to comply with 37 C.F.R.§1.72(b), requiring an abstract that will allow the reader to quicklyascertain the nature of the technical disclosure. The Abstract issubmitted with the understanding that it will not be used to interpretor limit the scope or meaning of the claims. In addition, in theforegoing Detailed Description, it can be seen that various features aregrouped together in a single embodiment for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter lies in less than allfeatures of a single disclosed embodiment. Thus the following claims arehereby incorporated into the Detailed Description, with each claimstanding on its own as a separate embodiment.

What is claimed is:
 1. A system comprising: a contextual informationmodule configured to determine contextual information, the contextualinformation including parameters associated with a device used inaccessing a website hosting a plurality of item listings; a qualityscore module executable by one or more processors and configured tocalculate a quality score for each item listing of the plurality of itemlistings, the quality score independent of attributes of a user andindependent of the contextual information, the quality score indicatinga probability of a user engagement with a respective item listing; aranking module configured to receive the contextual information and thequality scores and rank each of the plurality of item listings based onthe contextual information and the quality scores; and a recommendationmodule configured to automatically select one or more item listings ofthe plurality of item listings based on the ranking of the item listingsand provide information indicative of the selected one or more itemlistings.
 2. The system of claim 1, wherein the quality score module isfurther configured to determine the quality score using three or more ofitem listing freshness, item listing quality, item listing imagequality, cover image quality of the item listing, top five mean itemlisting quality listings, top five mean image quality of the itemlisting, number of item listings in the item listing, age of the itemlisting in the item listing, item listing last update time, followcounts of the item listing, unfollow counts of the item listing, sharecounts of the item listing, number of item listing views, and userengagement with the item listing.
 3. The system of claim 2, wherein thequality score module is further configured to utilize a random forestclassifier technique in determining the quality scores.
 4. The system ofclaim 1, wherein the recommendation module is configured to compare thedetermined quality scores to a threshold and wherein the item listingassociated with the quality score is only selected if the quality scoreis greater than the threshold.
 5. The system of claim 1, wherein thecontextual information includes a type of device the user is using toaccess the website including whether the device is a mobile device. 6.The system of claim 5, wherein: the ranking module is further configuredto pre-filter the item listings based on the contextual informationoffline; the ranking module is further configured to determine whetheruser attributes are available for the user and in response todetermining user attributes are available, further filter thepre-filtered item listings based on the user attributes and filter thefurther filtered item listings based on the determined quality scores tocreate a first filtered set of item listings and wherein therecommendation module is further configured to select an item listingfrom the first filtered set; and the ranking module is furtherconfigured to determine whether user attributes are available for theuser and in response to determining the user attributes are notavailable, further filter the pre-filtered item listings based on thequality scores to create a second filtered set and wherein therecommendation module is further configured to select an item listingfrom the second filtered set.
 7. The system of claim 5, wherein: theranking module is further configured to determine if user attributes areavailable and in response to determining user attributes are notavailable, the recommendation module is to select an item listingremaining after filtering the plurality of item listings based on thequality score and then filtering based on the contextual information;and the ranking module is further configured to determine if userattributes are available and in response to determining the userattributes are available, to recommend an item listing remaining afterfiltering the plurality of item listings based on the determined qualityscores and the user attributes, and then filtering based on thecontextual information.
 8. The system of claim 5, wherein: the rankingmodule is further configured to determine if user attributes areavailable and in response to determining user attributes are notavailable, to select an item listing remaining after filtering theplurality of item listings based on the contextual information thenfiltering based on the quality score; and the ranking module is furtherconfigured to determine if user attributes are available and in responseto determining the user attributes are available, to select an itemlisting remaining after filtering the plurality of item listings basedon user attributes and contextual information and then filtering basedon the determined quality scores.
 9. The system of claim 1, wherein theranking module is further configured to determine the quality scoresusing a logistic regression technique.
 10. The system of claim 1,wherein the ranking module is further configured to determine thequality scores using a stochastic gradient descent technique.
 11. Thesystem of claim 1, wherein the ranking module is further configured todetermine the quality scores using a Gaussian naïve Byes technique. 12.The system of claim 1, wherein the ranking module is further configuredto determine the quality scores using an Adaboost technique.
 13. Amethod comprising: determining, using a computer processor, contextualinformation, the contextual information including parameters associatedwith a device used in accessing a website hosting a plurality of itemlistings; determining a quality score for each of a plurality of itemlistings, the quality score independent of a user's attributes andindependent of the contextual information of the user, the quality scoreindicating a probability of a user engagement with a respective itemlisting; ranking each of the plurality of item listings based on thecontextual information and the quality scores; and automaticallyselecting one or more item listings of the plurality of item listingbased on the ranking of the item listings; and providing informationindicative of the selected item listing.
 14. The method of claim 13,wherein determining the quality scores includes determining the qualityscores using a random forest classifier technique, a logistic regressiontechnique, a stochastic gradient descent technique, Gaussian naïve Byestechnique, or an Adaboost technique.
 15. The method of claim 14, whereinranking each of the plurality of item listings includes: pre-filteringthe item listings based on the contextual information offline;determining whether user attributes are available for the user; inresponse to determining user attributes are available, further filteringthe pre-filtered item listings based on the user attributes andfiltering the further filtered item listings based on the determinedquality scores to create a first filtered set and selecting an itemlisting includes selecting an item listing from the first filtered set;and in response to determining the user attributes are not available,further filtering the pre-filtered item listings based on the qualityscores to create a second filtered set and wherein selecting an itemlisting includes selecting an item listing from the second filtered set.16. The method of claim 14, wherein ranking each of the plurality ofitem listings includes: determining if user attributes are available; inresponse to determining user attributes are not available, selecting anitem listing includes selecting an item listing remaining afterfiltering the plurality of item listings based on the quality score andthen filtering based on the contextual information; and in response todetermining the user attributes are available, selecting the itemlisting includes selecting an item listing remaining after filtering theplurality of item listings based on the determined quality scores andthe user attributes and then filtering based on the contextualinformation.
 17. The method of claim 14, wherein ranking each of theplurality of item listings includes: determining if user attributes areavailable; in response to determining user attributes are not available,selecting an item listing includes selecting an item listing remainingafter filtering the plurality of item listings based on the contextualinformation then filtering based on the quality score; and in responseto determining the user attributes are available, selecting an itemlisting includes selecting an item listing remaining after filtering theplurality of item listings based on user attributes and contextualinformation and then filtering based on the determined quality scores.18. A non-transitory machine-readable storage medium embodyinginstructions which, when executed by a machine, cause the machine toexecute operations comprising: determining contextual information, thecontextual information including parameters associated with a deviceused in accessing a website hosting a plurality of item listings;determining a quality score for each of a plurality of item listingsusing a random forest classifier technique, the quality scoreindependent of a user's attributes and independent of the contextualinformation of the user, the quality score indicating a probability of auser engagement with a respective item listing; ranking each of theplurality of item listings based on the contextual information and thequality scores; and automatically selecting one or more item listings ofthe plurality of item listing based on the ranking of the item listings;and providing information indicative of the selected item listing. 19.The non-transitory machine readable storage medium of claim 18, whereinthe instructions for ranking each of the plurality of item listingsinclude instructions, which when executed by the machine, cause themachine to perform operations comprising: determining if user attributesare available; in response to determining user attributes are notavailable, selecting an item listing includes selecting an item listingremaining after filtering the plurality of item listings based on thequality score and then filtering based on the contextual information;and in response to determining the user attributes are available,selecting the item listing includes selecting an item listing remainingafter filtering the plurality of item listings based on the determinedquality scores and the user attributes and then filtering based on thecontextual information.
 20. The non-transitory machine readable storagemedium of claim 18, wherein the instructions for ranking each of theplurality of item listings include instructions, which when executed bythe machine, cause the machine to perform operations comprising:determining if user attributes are available; in response to determininguser attributes are not available, selecting an item listing includesselecting an item listing remaining after filtering the plurality ofitem listings based on the contextual information then filtering basedon the quality score; and in response to determining the user attributesare available, selecting an item listing includes selecting an itemlisting remaining after filtering the plurality of item listings basedon user attributes and contextual information and then filtering basedon the determined quality scores.